Alternative missing data techniques to grade point average: Imputing unavailable grades

In this article, Grade Point Average (GPA) is considered a missing data technique for unavailable grades in school grade records. In Study 1, theoretical and empirical differences between GPA and 7 alternative missing grades techniques were considered. These 7 techniques are subject mean substitution, corrected subject mean, subject correlation substitution, regression imputation, EM algorithm imputation and two multiple imputation methods–stochastic regression imputation and data augmentation. The missing grade techniques differ greatly. Data augmentation and stochastic regression imputation appear to be superior as missing grades technique. In Study 2, the completed grade records (observed and imputed values) were used in two prediction analyses of academic achievement. One analysis was based on unweighed grades, the other on weighed grades. In both analyses, alternative missing grade methods produced better and more consistent predictions. It is concluded that some alternative missing grade methods are superior to GPA. The use of the Grade Point Average (GPA) as a measure of academic achievement has been discussed extensively in the literature. The GPA features often as a criterion in the prediction of academic performance and in the validation of college or university admission tests like SAT, ACT or GRE (e.g., Goldberg & Alliger, 1992; Nilsson, 1995; Stricker, 1991). In addition, person characteristics like personNiels Smits, SCHOLAR Schooling, Labor Market and Economic Development, Faculty of Economics, University of Amsterdam; Gideon J. Mellenbergh, Department of Psychological Methods, University of Amsterdam; Harrie C. M. Vorst, Department of Psychological Methods, University of Amsterdam, the Netherlands. We thank Conor V. Dolan and the reviewers for their valuable comments. Correspondence concerning this article should be addressed to Niels Smits, SCHOLAR, Faculteit der Economische Wetenschappen en Econometrie, Universiteit van Amsterdam, Roetersstraat 11, 1018 WB Amsterdam, the Netherlands. Electronic mail may be sent to [email protected]. ALTERNATIVE MISSING DATA TECHNIQUES TO GPA 2 ality traits or time management skills are also used to predict GPA (e.g., Britton & Tesser, 1991; Wolfe & Johnson, 1995). The GPA has also been used as an independent variable. It has been used as a predictor of success in higher education (e.g., Cariaga-Lo, Enarson, Crandall, Zaccaro, & Richards, 1997; House & Johnson, 1992) and of job performance after graduation (e.g., Bretz, 1989). It has been used both in psychological or educational research and in economic research (e.g., Betts & Morell, 1999). The appeal of the GPA is that it is well defined, widely understood, and easily obtainable from university records (Young, 1993). However, the use of GPA as a measure of performance does have its flaws. The GPA may differ between students in exact meaning. It is often based on ratings or grades obtained on a set of courses that differ in content from student to student. In practice, the GPA is treated as if it represents the same construct regardless of the student’s exact curriculum (e.g., Linn & Hastings, 1984). Variation in its exact meaning over students may decrease its reliability (e.g., Young, 1993). The widespread use of GPA is in part attributable to the fact that it is available for all students, even though variation in the composition of curriculum may render the averages not entirely comparable. The use of a more subject-specific measure of achievement would give rise to a missing data problem, because many students will not have followed the courses relating to the specific subject. Another problem in validation studies of pre-admission measures is the possible variation in grading standards within and between departments. Stricker, Rock, Burton, Muraki, and Jirele (1994), and Stricker, Rock, and Burton (1996) state that GPA lacks reliability and validity, because it cannot be used to compare students who take courses with severe grading standards and students who take courses with lenient standards. Several attempts have been made to adjust grades to render them more comparable (see, e.g., Young, 1990, 1993 ). Stricker et al. (1994) compared the effectiveness of several statistical methods to adjust GPA criteria for differences in grading standards of individual courses. One method involved the treatment of missing grades (missing because a student had not taken a given course), as a missing data problem. The unavailable grades were viewed as missing at random in the sense that they are predictable from the available grades. Stricker et al. (1994) pointed out that the EM algorithm may be used to obtain maximum likelihood estimates of the missing grades. Approaching the problem of unavailable grades as a missing data problem is not new. At least implicitly, this approach has often been taken. In calculating the GPA, the average grade for courses that a student has taken is implicitly substituted for missing grade scores. In this procedure, it is assumed that grades for each course are exchangeable, and that the data are missing completely at random. From this point of view, this method of calculating GPA can be considered as a simple missing data technique. A number of alternative techniques exist that may be applied to the missing grade problem. It should be noted that these techniques are not adjustments ALTERNATIVE MISSING DATA TECHNIQUES TO GPA 3 of GPA in the sense of Stricker et al. (1994), but missing data methods. In this article, the assumptions of GPA and alternative missing grade techniques are described and their use is demonstrated. This article includes two studies. In Study 1, theoretical and empirical similarities among the missing grade techniques are examined. In Study 2, the performance of these techniques is examined, when the imputed values are used as exact grades in the prediction of academic achievement among Dutch university students. We first provide a short description of the Dutch educational system, as it differs from the Anglo-Saxon system. Brief Outline of the Dutch Educational System In the Netherlands, primary education starts at age 4 and continues for 8 years to the age of 12. Following primary education, there are 4 levels of secondary education, of which one level is pre-university education (Dutch abbreviation: VWO) and lasts six years. At the end of secondary education, students take final examinations. Prior to the last two years of secondary education, students choose their examination subjects. Students in the VWO take exams in at least 7 of 15 subjects. These are Dutch (obligatory), English (obligatory), French, German, history, geography, two mathematics subjects, physics, chemistry, biology, two economics subjects, Latin and Greek. Students with a VWO certificate have automatic access to university education. They are not required to take additional admission tests. At university, master’s degree curriculum officially takes 4 years. Students are expected to earn 42 credits per annum (representing 42 weeks work). Lectures are given, but attendance in most courses is not obligatory. Students vary greatly in the number of credits they actually obtain in a year and in the total time they need to graduate. The amount of time they take to graduate does have financial consequences, however, because grants supplied by the Dutch government, to which students are entitled, are performance related. Students receive their grants initially in the form of a loan, which is converted to a non-repayable grant if they meet certain performance criteria. Study 1: Similarities and Differences between Missing Grades Techniques